Non-linear shrinkage estimation of large-scale structure covariance
نویسنده
چکیده
In many astrophysical settings, covariance matrices of large data sets have to be determined empirically from a finite number of mock realizations. The resulting noise degrades inference and precludes it completely if there are fewer realizations than data points. This work applies a recently proposed non-linear shrinkage estimator of covariance to a realistic example from large-scale structure cosmology. After optimizing its performance for the usage in likelihood expressions, the shrinkage estimator yields subdominant bias and variance comparable to that of the standard estimator with a factor of ∼50 less realizations. This is achieved without any prior information on the properties of the data or the structure of the covariance matrix, at a negligible computational cost.
منابع مشابه
Shrinkage Estimation of the Power Spectrum Covariance Matrix
We introduce a novel statistical technique, shrinkage estimation, to estimate the power spectrum covariance matrix from a limited number of simulations. We optimally combine an empirical estimate of the covariance with a model (the target) to minimize the total mean squared error compared to the true underlying covariance. We test our technique on N-body simulations and evaluate its performance...
متن کاملA shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics.
Inferring large-scale covariance matrices from sparse genomic data is an ubiquitous problem in bioinformatics. Clearly, the widely used standard covariance and correlation estimators are ill-suited for this purpose. As statistically efficient and computationally fast alternative we propose a novel shrinkage covariance estimator that exploits the Ledoit-Wolf (2003) lemma for analytic calculation...
متن کاملNonparametric Stein-type shrinkage covariance matrix estimators in high-dimensional settings
Estimating a covariance matrix is an important task in applications where the number of variables is larger than the number of observations. In the literature, shrinkage approaches for estimating a high-dimensional covariance matrix are employed to circumvent the limitations of the sample covariance matrix. A new family of nonparametric Stein-type shrinkage covariance estimators is proposed who...
متن کاملNonlinear shrinkage estimation of large-dimensional covariance matrices
Many statistical applications require an estimate of a covariance matrix and/or its inverse. Whenthe matrix dimension is large compared to the sample size, which happens frequently, the samplecovariance matrix is known to perform poorly and may suffer from ill-conditioning. There alreadyexists an extensive literature concerning improved estimators in such situations. In the absence offurther kn...
متن کاملNonlinear Shrinkage Estimation of Large-dimensional Covariance Matrices by Olivier Ledoit
Many statistical applications require an estimate of a covariance matrix and/or its inverse. When the matrix dimension is large compared to the sample size, which happens frequently, the sample covariance matrix is known to perform poorly and may suffer from ill-conditioning. There already exists an extensive literature concerning improved estimators in such situations. In the absence of furthe...
متن کامل